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Introduction 


Library  services  include  the  services  and  information  resources  provided  to  Alberta  Education  staff  by  the 
central  library  and  the  seven  decentralized  libraries:  the  five  regional  office  libraries  (Calgary,  Edmonton, 
Grande  Prairie,  Lethbridge  and  Red  Deer)  and  the  two  provincial  schools  (Alberta  Correspondence  School 
and  the  Alberta  School  for  the  Deaf). 

This  report  presents  the  findings  of  the  evaluability  assessment  of  Alberta  Education's  library  services. 
The  primary  purpose  of  this  procedure  is  to  ensure  credible  and  useful  evaluations.  An  evaluability 
assessment  determines  the  extent  to  which  a  program's  structure  is  amenable  to  an  effectiveness 
evaluation.  This  requires: 

•  the  program  to  be  clearly  defined  and  implemented  in  a  prescribed  manner; 

•  clearly  specified  objectives  and  effects;  and 

•  plausible  causal  linkages. 

The  evaluability  assessment  (EA)  is  the  first  phase  of  a  three-phase  evaluation  of  library  services.  The 
other  two  phases  are: 

Phase  Two:       an  external  effectiveness  evaluation  to  determine  the  manner 

and  extent  to  which  the  program  achieves  its  goals  or  produces  effects 

Phase  Three:     an  internal  review  of  the  program,  a  synthesis  of  the  results  of  the 
effectiveness  evaluation  and  the  review,  and  the  development  of  a 
long-term  (3  to  5  years)  action  plan  for  library  services 

The  EA  provides  a  plan  of  how  phases  two  and  three  will  proceed  and  what  they  will  include. 

The  evaluability  assessment  included  a  review  of  the  structure  of  library  services,  identification  of 
evaluation  issues  and  the  development  of  approaches  for  conducting  the  evaluation.  This  report  presents 
a  profile  and  description  of  library  services,  the  evaluation  issues,  the  options  for  evaluating  the  program 
and  the  recommendation  for  the  effectiveness  evaluation. 

The  EA  was  conducted  by  the  study  team  under  the  direction  and  guidance  of  Leonard  Rutman.  The  first 
two  days  of  training  (June  4  and  5)  provided  the  group  with  an  orientation  to  and  overview  of  EA.  The 
study  team  was  trained  in  the  development  of  a  documents  model  and  subsequently  prepared  one  for 
library  services.  The  central  library  manager  was  interviewed  to  clarify  the  documents  model  and  to  aid  in 
identifying  evaluation  issues. 

Between  the  first  and  second  training  sessions,  the  project  coordinator  interviewed  representatives  from 
the  regional  offices,  branches  and  the  provincial  schools  in  order  to  clarify  the  emerging  program  model 
and  to  identify  additional  evaluation  issues.  During  the  second  training  session  (June  13  and  14),  the 
program  model  and  evaluation  issues  were  reviewed.  Priorities  were  assigned  to  the  evaluation  issues  by 
the  director  of  the  Financial  and  Administrative  Services  Branch.  These  issues  were  then  organized  into 
four  evaluation  options  to  examine  the  program,  its  delivery,  potential  new  services  and  the  optimal 
technological  configuration  for  library  services.  For  each  of  these  four  options,  the  study  team  suggested 
appropriate  methodological  approaches. 
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This  chapter  presents  a  profile  of  library  services.  It  includes  the  background,  the  mandate,  the 
organization  and  the  allocated  resources. 


[i3®(gfe^Dr®[!i]irD(£l 

Alberta  Education's  central  library  dates  to  1971  when  it  was  developed  to  provide  a  resource  base  for 
departmental  staff.  Over  the  years,  the  services  have  expanded  to  include  the  department's  published 
documents  and  research  reports,  and  specialized  reference  collections.  Cataloguing  of  professional 
resources  was  done  by  the  central  library. 

As  the  curricular  resource  collections  in  the  regional  offices  expanded,  the  Key  Word  out  of  Context 
(KWOC)  system  for  organizing  these  resources  was  implemented  In  1977.  This  resulted  in  four  of  the 
regional  offices  (Edmonton,  Grande  Prairie,  Red  Deer  and  Calgary)  hiring  library  technicians  to  organize 
and  maintain  their  collections. 

In  1979,  the  central  library  acquired  a  terminal  and  modem  and  began  accessing  commercial  databases 
(DIALOG  and  ORBIT)  for  reference  information;  the  central  library  currently  accesses  16  systems. 
Departmental  requests  for  on-line  searches  are  coordinated  through  the  central  library.    Utilizing  the 
technological  advances  to  identify  and  retrieve  information  has  brought  considerable  change  to  the  central 
library's  ability  to  provide  reference  and  information  services  to  the  department. 

In  1982,  the  central  library  signed  a  contract  with  UTLAS  (University  of  Toronto  Library  Automation  System) 
to  develop  and  maintain  an  information  database  of  resources  acquired  and  produced  by  the  department. 
This  initiative  resulted  in  providing  access  to  departmental  resources  by  all  Alberta  Education  staff. 
Developing  the  department's  UTLAS  database  required  that  all  existing  resources  in  the  collection  be 
retrospectively  converted  to  MARC  (machine  readable  cataloguing  )  and  AACR  II  (Anglo-American 
Cataloguing  Rules  II)  format  and  use  of  the  Library  of  Congress  classification  system  so  that  resources 
would  be  accessible  to  all  subscribers  on  the  participating  system.  Thus  began  a  period  of  intensive 
conversion  of  the  existing  manual  cataloguing  system  to  the  automated  system  of  the  department's 
information  resources.  An  inventory  was  completed  of  all  professional  resources  located  in  the  central 
library,  the  five  regional  office  resource  centres  and  the  Alberta  School  for  the  Deaf.  The  conversion  of  the 
curricular  resources  indexed  in  the  KWOC  system  began  in  January  1983  and  is  partially  completed. 
Since  the  introduction  of  automation  in  1982,  the  Alberta  School  for  the  Deaf  and  the  Alberta 
Correspondence  School  collections  are  being  organized  by  the  central  library. 

In  July  1983,  the  central  library  acquired  the  INNOV ACQ  On-line  Acquisition  System.  This 
microcomputer-based  system  interfaces  with  UTLAS  and  maintains  the  order  files  for  books,  periodicals, 
and  the  curriculum  evaluation  resources  ordered  through  the  central  library. 

The  changes  in  the  acquisitions,  cataloguing  and  reference  services  have  resulted  in  a  need  to  examine 
the  new  roles  and  responsibities  of  the  department's  library  staff  and  resources,  and  to  identify  the  ways 
that  the  service  to  the  department  can  be  improved  so  that  all  staff  members  have  equal  access  to  the 
potentially  limitless  resources  of  the  global  network.  Examination  of  the  service  itself  will  provide 
information  to  assist  in  making  decisions  about  the  direction  that  library  services  should  take  within  the 
department. 
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The  mandate  for  library  services  was  stated  in  the  March  1 , 1977  memo  from  the  Deputy 

Minister  to  the  Associate  Deputy  Ministers,  Directors  and  Responsibility  Centre  Heads.  The  policy 

established  by  Directors'  Council  indicated  that: 

All  materials  purchased  with  government  funds  were  to  be  inventoried  by  the  library 
or  resource  centre  and  could  then  be  kept  in  an  employee's  office  on  an  extended  loan  basis 
if  necessary. 

•  All  materials  donated  to  the  employee's  position  and  intended  to  be  used  for  the 
instruction  of  children  or  intended  by  the  publishing  company  to  be  used  by  the  employee 
for  inservice,  workshop  or  display  purposes  were  to  be  considered  the  property  of  the 
Government.  These  materials  were  to  be  reviewed  by  the  employee  as  to  their 
suitability  before  placing  them  in  the  library  or  resource  centre  for  cataloguing. 
Periodic  reviews  of  such  materials  were  to  be  carried  out  by  the  employee  to  keep  the 
collection  relevant. 

•  Within  Alberta  Education,  any  materials  acquired  under  the  above  two  clauses  were  to 
be  made  available  to  any  employee  of  the  department  through  the  library  or  resource 
centre. 

•  Employees  who  requested  the  searching  for  information  and/or  the  collecting  of  items 
identified,  for  a  use  not  connected  directly  with  their  work  for  Alberta  Education,  were  to 
submit  a  request  on  a  form  ahead  of  time  to  the  librarian  who  would  make  a  decision  in 
consultation  with  the  Associate  Deputy  Minister  (Support  Services)  as  to  the  legitimacy 
of  that  request. 

The  curriculum  materials  database  was  established  to  implement  the  mandate,  approved  by  Directors' 
Council  November  6, 1977,  for  purchased  and  donated  resources  located  in  the  regional  offices.  The  Key 
Word  Out  of  Context  (KWOC)  indexing  program  was  to  be  used  to  record  and  provide  access  to  these 
curriculum  materials.  The  change  from  the  KWOC  to  the  UTLAS  system  using  Library  of  Congress 
classification  was  negotiated  with  the  regional  office  directors  and  consultants. 

The  expansion  of  the  central  library's  services,  such  as  the  acquisition  of  authorized  curricular  resources 
and  the  development  of  the  curriculum  evaluation  database,  has  resulted  from  the  automation  of  services 
and  was  formalized  through  functional  relationship  statements  with  the  respective  branches. 

Alberta  Education  was  reorganized  in  1983  to  decentralize  decision  making  and  to  make  it  more 
responsive  to  local  needs.  Subsequent  initiatives  such  as  the  Management  and  Finance  Plan  and 
the  Secondary  Education  Programs  Review  have  also  changed  the  roles  of  departmental  staff 
whose  information  needs  may  change  as  a  result  of  these  new  responsibilities. 


The  eight  departmental  libraries  (central,  five  regional  offices.  Alberta  Correspondence  School  and 

Alberta  School  for  the  Deaf)  report  to  different  divisions  within  the  department.  The 

central  library  is  located  in  the  Financial  and  Administrative  Services  Branch  of  the  Finance  and 

Administration  Division  while  the  remaining  libraries  report  to  their  regional  office  or  to 

the  school;  regional  offices  and  provincial  schools  are  located  in  the  Delivery  Division. 
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Figure  1  presents  the  departmental  library  network.  All  libraries  interact  with  the  central  library  and  among 
themselves.  The  communication  among  the  libraries  is  done  verbally  by  telephone,  in  writing  through 
the  government  courier,  or  on-line  to  the  central  library's  acquisition  system.  The  decentralized  libraries 
can  access  the  resources  in  the  other  departmental  libraries  by  using  the  book  and  comfiche  catalogues; 
they  can  access  book  and  periodical  order  files  by  accessing  the  central  library's  acquisition  system  either 
on-line  or  from  printouts.  Only  the  central  library  has  on-line  access  to  external  database  systems. 

The  central  library  currently  completes  the  acquisition  and  cataloguing  for  the  decentralized  libraries  which 
use  the  services.  Once  the  resources  are  catalogued,  they  are  sent  to  the  library  which  ordered  them  and 
the  resources  are  listed  in  book  and  comfiche  catalogues.  Each  library  is  responsible  for  organizing  and 
maintaining  its  collection. 

Periodicals  are  ordered  through  the  central  library's  automated  acquisition  system  and  are  processed  and 
routed  to  the  departmental  libraries.  These  periodicals  are  listed  in  the  weekly  contents  publication  and 
sent  to  the  professional  and  managerial  staff  requesting  this  service.  Requests  for  articles  are  filled  by  the 
library  holding  the  periodical. 

Information  requests  by  departmental  staff  are  directed  to  their  local  librarian  who  can  assist  in  locating  and 
retrieving  the  desired  resource  in  Alberta  Education's  libraries,  or  from  other  government  libraries,  local 
educational  institutions  and  public  libraries.  Resources  from  these  other  libraries  can  be  borrowed 
through  interlibrary  loan.  Staff  can  also  be  directed  to  the  central  library  if  the  desired  resource  is  not 
available  from  these  libraries.  The  central  reference  librarians  search  for  the  desired  item  by  accessing 
information  networks  (manual  and  on-line)  and  then  retrieve  it  for  the  staff  member. 


For  the  fiscal  year  ended  March  31 , 1985,  Alberta  Education  spent  $1 .38  million  for  library  services  and 
information  resources.  This  figure  includes: 

•  the  cost  of  the  central  library  operation; 

•  the  start-up  costs  for  the  Alberta  Correspondence  School  library; 

•  640  expenditures  for  regional  offices,  branches  and  provincial  schools*; 

•  the  cost  of  staff  to  maintain  the  decentralized  library  collections. 

Subtracting  the  start-up  expenditures  for  the  Alberta  Correspondence  library  leaves  an  annual  library 
services'  operating  cost  and  purchase  of  information  resources  within  the  department  at  approximately 
$1.1  million. 

The  central  library  employs  12  full-time  staff  while  the  decentralized  libraries  (the  regional  offices,  ACS  and 
ASD)  employ  an  additional  7  full-time  equivalents  for  a  total  of  19  full-time  equivalent  staff  members. 


*640  expenditure  codes  for  the  Alberta  Correspondence  School  and  the  Alberta  School  for  the  Deaf 
include  student  textbook  materials  in  addition  to  library  materials. 
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Figure  1 
Departmental  Library  Network 
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Description 


This  chapter  presents  the  library  services  model  which  was  developed  through  analysis  of  the  documents, 
interviewing  Alberta  Education  personnel  and  examining  the  program  in  operation. 
Figure  2  presents  the  resulting  library  services  model  which  consists  of  components,  outputs  and 
objectives/effects.  The  following  definitions  apply  to  the  model: 

•  A  program  is  a  group  of  resources  and  activies,  and  their  related  direct  outputs, 
undertaken  pursuant  to  a  given  external  objective  or  set  of  related  objectives  and 
administered  by  a  department  or  agency  of  government. 

•  A  component  is  a  subset  of  the  program  which  can  be  linked  to  external 
objectives/goals  and  is  a  suitable  focus  for  evaluation. 

•  Outputs  are  the  goods  and  services  produced  by  program  personnel  and  distributed 
outside  the  program  organization. 

•  Objectives  are  purpose  statements  which  indicate  what  is  to  be  accomplished  in  terms 
of  impacts  and  effects,  not  outputs  or  work  processes. 

•  Effects  are  the  intended  and  unintended  consequences  of  a  program  including  formally 
stated  objectives. 

There  are  five  components  in  the  library  services  model:  collection,  databases,  network,  reference  ( all 
four  are  interrelated)  and  audiovisual  equipment.  Each  component  produces  at  least  one  output.  The 
outputs  for  each  component  are: 


Collection 

specialized 

professional 

curricular 


Databases 

UTLAS 

catalogues 

specialized 


Network 


Reference 


uses  by  AE  information 
uses  by  others 


Audiovisual 
Equipment 

types 


The  specific  objectives  of  the  four  interrelated  components  are  to: 


•  optimize  information  sharing 

•  optimize  utilization 

•  reduce  duplication  and 

•  increase  time  efficiency 


All  five  components  exist  to  meet  users'  needs. 


The  elaboration  of  the  model  describes  more  fully  the  program's  components,  outputs  and 
objectives/effects  and  the  linkages  among  them. 
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The  collection  contains  information  resources  which  are  frequently  used  in  the  department.  The  major  part 
of  the  collection  is  located  in  the  central  library  with  smaller  parts  decentralized  to  the  regional  offices, 
provincial  schools  {  Alberta  Correspondence  School  and  the  Alberta  School  for  the  Deaf)and  some 
branches;  the  decentralized  resources  are  required  to  meet  the  unique  needs  of  staff  in  these  locations. 


OUTPUTS 


OBJECTIVES/EFFECTS 


Specialized  Collection 

AE  published  documents 
Legislative  Acts 

Alberta  and  other  provinces 
Reference  resources 
ERIC 

Micro  media 

Educational  Testing  Service 
Publishers'  catalogues 
Selection  aids 

bibliographic  verification 

identification 

books 

periodicals 

on-line  databases 


Optimize  information  sharing 

degree  of  reliance  on  other 
sources  of  information 

Optimize  utilization 

frequency  of  use 

Reduce  duplication: 

resources 
user  time 
staff  time 

Increase  time  efficiency 

user  time 


Professional  Collection 

Books 

Monographs 
Periodicals 
AV  resources 


Meet  Users'  Needs 

availability  of  information 
ease  of  access 
appropriateness  of  material 
speed  in  meeting  request 
need  for  service 


Curricular  Collection 


Authorized  curriculum  resources 
Authorized  courseware 
Curriculum  resources  in 

evaluation  process 

pilot  process 
Curriculum  guides 

Alberta  and  other  provinces 

current  and  historical 
Diploma  exams 

answer  keys 

analysis  of  results 
Curriculum  development  library 
Special  education  resources 
School  library  resources 

Alberta  Correspondence  School 

Alberta  School  for  the  Deaf 
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The  databases  list  the  information  resources  and  their  locations  within  the  department.  Manual  access  to 
the  databases  for  ail  regional  offices,  provincial  schools  and  branches  is  through  the  print  and  comfiche 
catalogues  which  are  produced  on  a  quarterly  basis.  The  central  library  accesses  the  database  on-line 
through  UTLAS  or  from  in-house  databases  for  the  most  up-to-date  information  on  the  department's 
collection. 


OUTPUTS 


OBJECTIVES/EFFECTS 


Database  (UTLAS  on-line) 

Departmental  resources  listed  in 
internationally  standardized  MARC  and 
AACR  II  format: 

specialized  materials 

books 

monographs 

periodicals 

curriculum  resources 
Library  and  branch  location  of  AE  resources 
Reference  services 

on-line  identification 

location  and 

retrieval  of  resources  from  participating 
libraries  and  research  institutions  on  UTLAS 
Bibliographic  verification  for  acquisition 
Departmental  libraries  participating  in  UTLAS 


Optimize  information  sharing 

degree  of  reliance  on  other 
sources  of  information 

Optimize  utilization 

frequency  of  use 


Reduce  duplication: 

resources 
staff  time 
user  time 


Increase  time  efficiency 

user  time 
staff  time 


Catalogues  (print  and  comfiche) 

Lists  of  AE  monthly  acquisitions 
Listing  of  location  of  AE  holdings 
on  quarterly  basis 
Resources  listed  under: 

title 

series 

subject 

curriculum  course  name 
author 

Library  of  Congress 
Specialized  collection  and  location 
authorized  curriculum  resources 
Alberta  Correspondence  School 
Alberta  School  for  the  Deaf 

Specialized  Databases 


Meet  users'  needs 

reduce  need  to  purchase 
reduce  cost  to  department 
facilitiate  identifying  resources 
increase  access  to  resources 


On-line  identification  and  retrieval  of: 
AE  documents  and  print  catalogues 
monthly  acquisition  listings 
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Databases  (continued) 

Legislative  acts  listings 
Reference/referral  listing  of  requests  by 
subject 
branch 

tracking  of  interlibrary  loans 
Professional  development 
user  information  profiles 
SDI  (selected  dissemination  of  information) 
scanning  and  monitoring 


The  network  is  the  method  whereby  the  department  has  access  to  Information  resources  beyond  those 
of  its  own  collection.  By  participating  in  a  cooperative  system,  Alberta  Education  can  obtain  resources  from 
provincial,  national  and  international  members  of  the  system.  This  provides  staff  with  potential  access  to 
information  resources  from  virtually  anywhere  in  the  world. 


OUTPUTS 

Uses  by  AE  of  others'  resources 
Contract  for  use  of  services 

Commercial  databases 

1 6  with  50  files  each  for  a 
total  of  800  files 

Research  databases 

UTLAS  (300  files) 

Participants  in  cooperative  system 

Departmental 

UTLAS  departmental  file 
Provincial 

ASIST  (ABS) 

government  libraries 

college  libraries 

U  of  A  education  library 

educational  institutions 
National 

provincial  departments  of 
education  and  Canadian  libraries 
via  UTLAS,  INET,  DOBIS 

research  institutions 

Micromedia 

National  Library  of  Canada 
International 

ERIC  Clearinghouse 


OBJECTIVES/EFFECTS 

Optimize  information  sharing 

degree  of  reliance  on  other 
sources  of  information 


Optimize  utilization 

frequency  of  use 


Reduce  duplication: 

resources 
staff  time 
user  time 


Increase  time  efficiency 

user  time 
staff  time 


IVIeet  users'  needs 

most  current  information 
availability  of  global  resources 
equal  access  to  all  departmental 
collections 
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Network  (continued) 

Document  retrieval/electronic 
messaging 

Contract  for  use  of  sen/ice 
ENVOY 
UTLAS 

CANOLE  (CISTI) 
INET 
ERIC 

Participant  in  cooperative 
system 

Government  courier 

RITE 

Mail 

Uses  by  others  of  AE's  resources 

The  same  uses  described  under  participants 
in  cooperative  system  above  for  AE  staff  since 
network  members  share  resources  where  possible. 


Specialized  library  personnel  provide  reference  services  to  assist  departmental  staff  in  locating  and 
retrieving  information  resources  required  to  meet  their  unique  needs.  Through  the  interactive  capability  of 
the  network,  it  is  possible  to  identify  the  most  current  information  available  on  a  specific  topic  area  in  a  quick 
and  efficient  manner;  this  reference  service  frees  departmental  staff  from  searching  tasks  so  they  can 
concentrate  on  interpreting  the  information. 


OUTPUTS 


OBJECTIVES/EFFECTS 


Information 


Optimize  information  sharing 

locate  and  retrieve  existing 
information 


Types 


Manual 
On-line 


Optimize  utilization 

reuse  of  identified  and  retrieved 
information 


bibliographic 
full  text 
numeric 


Quality  of  each  type 


Reduce  duplication: 


Timely 
Current 


resources 
staff  time 
user  time 
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Reference  (continued) 


Specific  to  topic 
Useful 

Accessible  (retrievable) 


Increase  time  efficiency 

coordinate  users'  requests 
reduce  turnaround  time 


Users  of  Information 


Meet  users'  needs 


Regional  offices 
Central  office 
Administrators 
Financial  personnel 
Planners 
Evaluate  rs 

Curriculum  developers 
Researchers 

Learning  resources  officers 
Delivery  personnel 


reduce  user's  search  time 
more  effective  access 

to  quality  information 
facilitate  identification  of 

information 

locate  available  information 
efficient  information  retrieval 
Identification  of  new  sources  of 
information 

referral  to  other  research 
sources 


Teachers  (ACS,  ASD) 
Research  groups 
Ad  hoc  committees 
External  users 


^[lJ](o]0®wa^3J]ilQ  l(o][!i]O^DnJ]©QfDa 

Audiovisual  equipment  is  maintained  in  the  central  library  for  the  use  of  departmental  staff  in  examining 
audiovisual  resources  so  that  they  can  be  evaluated  and  appropriate  ones  acquired.  This  also  provides 
staff  with  functioning  equipment  which  can  be  used  in  making  audiovisual  presentations. 


OUTPUTS 


OBJECTIVES/EFFECTS 


Types  of  Equipment 


Meet  users'  needs 


1 6  mm  projectors 
VTRs  and  monitors 
filmstrip  projectors 
overhead  projectors 
other  AV  equipment 


facilitate  evaluating  resources 
facilitate  presentations 


increase  efficiency 
minimize  breakdown 
decrease  frustration 


reduce  duplication  of  services 


Issues 
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The  second  step  in  evaluability  assessment  is  to  identify  evaluation  issues.  This  was  accomplished 
through  interviews  with  the  central  library  manager  and  Alberta  Education  staff  members  from  the  regional 
offices,  branches  and  provincial  schools.  More  than  25  people  were  asked  to  identify  evaluation  issues. 

These  issues  were  then  compiled  by  the  study  team  and  assigned  to  one  of  four  types:  program 
rationale,  program  effectiveness,  program  design  and  delivery,  and  alternatives.  Three  types  of  action 
were  recommended:  no  action,  an  effectiveness  evaluation  to  be  undertaken  in  phase  2  of  the 
evaluation,  or  a  management  review  to  be  undertaken  during  phase  3  of  the  evaluation.  The  reason  a 
particular  type  of  action  was  recommended  for  each  issue  is  included. 


Issues 


Type 

No 

Action 
(NA) 


of  Action  Recommended 


Effectiveness  Management 
Evaluation  Review 
(EE)  (MR) 


Reason 


Phase  2         Phase  3 


Importance  of 
each  component 


All  components  are  considered 
essential.  EE  will  indicate  extent  to 
which  services  are  meeting  users' 
needs. 


Adequacy  and 
distribution  of 
resources  (dollars 
and  staff) 


EE  of  other  issues  and  MR  of  this 
issue  should  identify  need  for 
changes  in  resources. 


Relationship  among 
information  units 
(CIS,  CSDBM, 
RS,  CB)* 


Management  can  address  this  issue 
at  any  time. 


*  CIS  (Corporate  Information  Systems);  CSDBM  (Computer  Systems  and  Data  Base 
Management);  RS  (Records  Services);  CB  (Communications  Branch). 
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Issues 


Type  of  Action  Recommended 

No       Effectiveness  Management 
Action    Evaluation  Review 
(NA)         (EE)  (MR) 
Phase  2         Phase  3 


Reason 


Meeting  users'  needs 
through  each  component 


EE  will  demonstrate  quality  of 
service. 


Achieving  objectives 
of  each  component 


EE  will  demonstrate  quality  of 
service. 


Who  uses  the 
library  (user)? 


This  is  related  to  the  location;  use 
could  influence  funding. 


What  are  the  uses 
of  the  library? 


EE  will  illuminate  uses;  it  will  build 
on  uses  by  specific  users. 


Appropriateness 
of  locations 


EE  will  determine  whether 
arrangement  results  in  appropriate 
sen/ices  for  regions  (ROEs  and 
ACS,  ASD). 


Appropriateness  of 

organizational 

arrangement 


EE  will  address  the  efficiency  and 
coordination  issues. 


Efficiency  of  LS 


MR  (phase  3)  an  operational  audit 
could  identify  the  costs  and  the  best 
value  for  the  available  funds. 
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Issues 


Type  of  Action  Recommended 

No        Effectiveness  Management 
Action    Evaluation  Review 
(MA)  (EE)  (MR) 

Phase  2         Phase  3 


Reason 


Centralization/ 
decentralization 


EE  will  indicate  the  quality  of 
sen/ices  to  ROEs  and  ACS.  ASD. 


Cost  savings 


MR  (phase  3)  will  assess  the  need 
and  feasibility  of  cost  savings.  EE 
will  help  shed  light  on  the  issue. 


Greater  reliance 
on  network  to 
provide  information 


MR  (phase  3)  will  make  policy 
choice;  opportunity  for  cost 
savings. 


New  services 


EE  will  identify  new  services 
considered  to  meet  users'  needs 
better. 


Optimal  technological 
configuration 


EE  will  determine  how  to  enhance 
the  effectivess  of  library 
services;  OTCwillbe 
determined  in  phase  3. 


Options 
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Once  the  issues  had  been  identified,  they  were  reviewed  to  determine  which  ones  should  be  included  in 
the  external  effectiveness  evaluation  to  be  undertaken  during  phase  2  of  the  evaluation.  Those 
considered  to  have  top  priority  were  organized  into  options  for  further  investigation. 

The  four  options  considered  to  have  top  priority  included  the  issues  related  to  the  program,  the 
delivery  of  the  program  and  two  of  the  alternatives  --new  services  and  optimal  technological 
configuration.  The  first  two  options  capture  the  main  concerns  of  the  staff  members  who  were  interviewed. 
The  last  two  options  are  concerned  with  the  potential  for  enhancing  the  service  for  Alberta  Education  staff. 
The  suggested  approach  for  each  as  well  as  the  questions  to  be  addressed  are  reported  for  each 
evaluation  option  separately. 


The  first  option  is  an  evaluation  of  the  program  itself.  The  suggested  approach  involves  three  major 
activities:  examination  of  library  documents;  development  of  a  questionnaire  to  be  administered  to  all 
professional  and  managerial  staff  in  Alberta  Education;  and  interviewing  a  sample  of  staff  members  from 
the  regional  offices,  branches  and  provincial  schools.  The  key  questions  and  the  suggested  method  for 
answering  them  are  listed  below. 


Suggested  Approach 

Program  Questionnaire  Interview  Documents 

all  sanple  library 

professional/managerial  staff 


1 .1     Who  are  the  users  of  AE  libraries? 

1.1.1  central 

1.1.2  regional  office 

1 .1 .3  Alberta  Correspondence  School 

1 .1 .4  Alberta  School  for  the  Deaf 

1.1.5  branch 


1 .2     What  are  the  uses  of  the  library? 

1.2.1  reference 

1.2.2  resources 

1.2.3  referral 

1.2.4  orientation 

1.2.5  inservice 
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Suggested  Approach 

Program  Questionnaire  Interview  Documents 

all  sanple  Ibrary 

professional/managerial  staff 


1 .3     To  what  extent  are  users'  needs  met 
in  the  departmental  libraries? 

1.3.1  central 

1.3.2  regional  office 

1 .3.3  Alberta  Correspondence  School 
^   1 .3.4  Alberta  School  for  the  Deaf 

1.3.5  branch 


1 .4     How  does  each  component  contribute 
to  users'  needs? 

1.4.1  collection 

1.4.2  database 

1 .4.3  network 

1 .4.4  reference 

1 .4.5  audiovisual  equipment 


1 .5     To  what  extent  are  the  immediate 
objectives  met? 

1 .5.1  optimize  information  sharing 

1.5.2  optimize  utilization 
1 .5.2     reduce  duplication 

1 .5.4     increase  time  efficiency 


1 .6     How  does  each  component  contribute 
toward  achieving  the  objectives? 


1 .7     Which  users  derive  the  greatest/least 
benefit? 
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The  second  option  consists  of  examining  the  organization  of  library  services,  that  is,  how  the  services  are 
delivered.  The  suggested  approach  includes  using  the  questionnaire  developed  for  option  one, 
interviewing  all  library  services  (LS)  staff  in  the  central  and  decentralized  libraries,  and  examining  library 
and  database  documents. 


Suggested  Approach 

Delivery  Questionnaire*  Interview  Documents 

all  library 
LS  staff  database 


2.1     How  are  components  implemented  in  the 
central,  regional  office,  provincial  school 
(ACS  and  ASD)  and  branch  libraries? 

2.1.1  collection 

2.1.2  database 

2.1.3  network 

2.1.4  reference 

2.1 .5  audiovisual  equipment 


2.2  To  what  extent  is  there  appropriate  or 
inappropriate  duplication  of 
holdings  in  departmental 
libraries? 

2.2.1  central 

2.2.2  regional  office 

2.2.3  Alberta  Correspondence  School 

2.2.4  Alberta  School  for  the  Deaf 

2.2.5  branch  collections 


2.3     How  are  services  implemented  in  the 
central,  regional  office,  provincial 
school  (ACS  and  ASD)and  branch 
libraries? 

2.3.1  acquisitions 

2.3.2  cataloguing 

2.3.3  information  and  reference  services 

2.3.4  maintenance  of  collections 


The  questionnaire  for  professional  and  managerial  staff. 
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Suggested  Approach 

Delivery  Questionnaire*  Interview  Documents 

all  library 
LS  staff  database 


2.4     Are  there  gaps  in  services? 


2.5     Are  services  coordinated 

between  the  central  and  other 
AE  libraries  and  branches? 

2.5.1  acquisitions 

2.5.2  cataloguing 

2.5.3  information  and  reference  services 

2.5.4  maintenance  of  collections 


2.6     Are  staff  referred  to  the  other 

information  units  when  appropriate? 


2.7     Is  there  inappropriate  and/or  costly 
duplication  of  services? 

2.7.1  acquisitions 

2.7.2  cataloguing 

2.7.3  information  and  reference  services 

2.7.4  maintenance  of  collections 


2.8     Who  is  responsible  for  library  services 
in  the  central,  regional,  provincial  school 
and  branch  libraries? 

2.8.1  acquisitions 

2.8.2  cataloguing 

2.8.3  information  and  reference  services 

2.8.4  maintenance  of  collections 


2.9     Does  the  central  library  meet  regional 
library  needs  in  a  timely  way? 


The  questionnaire  for  professional  and  managerial  staff. 
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The  third  option  is  concerned  with  identifying  new  services  which  the  libraries  might  provide.  The  method 
of  determining  the  need  and  anticipated  benefits  of  potential  new  services  includes:  asking  professional 
and  managerial  staff  these  questions  through  the  questionnaire;  interviewing  the  LS  staff,  as  well  as 
recognized  library  experts;  and  examining  library  documents  and  the  literature. 


Suggested  Approach 

New  Services  Questionnaire*    Interview  Documents 

LS  staff  library 
library  experts  literature 


3.1  Is  there  a  need  for  the  following 
services? 

3.1 .1  scanning  and  monitoring 
publications  to  notify  users 

3.1.2  processing/interpreting 
information 

3.1.3  reformatting  bibliographies 
to  meet  specific  requests 

3.1 .4  staff  orientation 

3.1 .5  on-line  access  by  regional/school 
libraries  to  databases 

3.1 .6  communication  link  between 
user  and  service 

3.1.7  other  services 

3.2  What  are  the  benefits  of  these 
services? 

3.2.1  scanning  and  monitoring 
publications  to  notify  users 

3.2.2  processing/interpreting 
information 

3.2.3  reformatting  bibliographies 
to  meet  specific  requests 

3.2.4  staff  orientation 

3.2.5  on-line  access  by  regional/school 
libraries  to  databases 

3.2.6  communication  link  between 
user  and  service 

3.2.7  other  services 


The  questionnaire  for  professional  and  managerial  staff. 
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The  final  option  is  to  determine  the  optimal  technological  configuration  for  library  services.  The  suggested 
approach  for  this  option  is  to  engage  a  library  automation  expert  to  assist  with  this  task.  A  literature 
review  of  this  area  would  also  be  helpful  in  determining  the  optimal  technological  configuration  of  library 
services. 


Suggested  Approach 

Optimal  Technological 

Configuration  Library  Automation  Literature 

Expert  Review 


4.1  To  what  extent  should  technology  be  used  to 
identify  information  sources  (databases)? 

4.2  Should  there  be  a  circulation  module  on  the 
system  at  regional  office  and  provincial 
school  libraries  for  tracking  resources? 

4.3  Should  there  be  on-line  access  from  regional 
and  provincial  school  libraries  to  facilitate 
equal  access  to  all  users? 

4.4  What  would  constitute  the  most  cost- 
effective  organizational  arrangement? 


Recommendation 


The  recommendation  of  the  study  team  is  to  select  options  1 ,  2  and  3  for  the  external  effectiveness 
evaluation  in  phase  2  of  the  evaluation  of  library  services.  These  three  options  can  all  be  undertaken  by 
the  same  consultant  or  consulting  firm.  It  is  anticipated  that  the  elapsed  time  to  complete  the  effectiveness 
evaluation  would  be  approximately  six  months.  Investigation  of  the  recommended  options  would  be 
possible  given  the  fiscal  constraints  on  the  evaluation. 

The  decision  to  conduct  the  final  option,  that  of  determining  the  optimal  technological  configuration  of 
library  services,  is  deferred  until  phase  3.  Once  the  effectiveness  of  the  program  and  its  delivery  are 
known,  management  will  be  in  a  better  position  to  ascertain  the  need  to  enhance  the  technological 
capability  of  the  libraries. 


Appendix 
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Wholey  (1977)  and  his  colleagues  at  The  Urban  Institute  pioneered  an  approach  called  evaluability 
assessment  (EA)  for  carrying  out  a  pre-evaluation  analysis  of  a  program.  Rutman  modified  and  adapted  the 
EA  for  planning  useful  evaluations. 

Evaluability  assessment  is  a  procedure  entailing  the  analysis  of  documents  and  the  conduct  of  interviews 
in  order  to  develop  a  model  of  a  program's  components,  outputs,  objectives  and  effects.  The  EA  provides 
some  assurance  that  relevant  variables  are  surfaced  for  inclusion  in  the  evaluation.  It  also  provides  an 
opportunity  to  develop  collaborative  relationships  with  administrators  and  program  staff. 

The  steps  in  carrying  out  an  evaluability  assessment  are: 

1 .  Review  program  structure 

1 .1  prepare  a  program  documents  model 

1.1.1  list  program  components  and  objectives/effects 

1 .1 .2  prepare  a  flow  model 

1 .2  interview  program  personnel 

1.2.1  select  who  should  be  interviewed 

1 .2.2  explain  the  EA  procedure 

1 .2.3  verify  program  documents  model 

1 .2.4  determine  interviewees'  views  of  the  program 

1 .3  undertake  field  work  to  examine  program  operations 

1 .3.1  determine  how  the  program  is  implemented 

1 .3.2  identify  the  people  served  by  the  program 

1.3.3  verify  objectives/effects 

1 .4  develop  evaluable  program  model 

2.  Identify  evaluation  issues 

2.1  review  program  documents 

2.2  interview  major  stakeholders 

2.3  identify: 

2.3.1  dimensions  of  the  issues 

2.3.2  rationale  for  studying  issues 

2.3.3  priority  attached  to  each  issue  by  decision  makers 

2.4  place  related  issues  under  common  theme 

3.  Develop  evaluation  options 

3.1  cluster  issue  groups  into  options 

3.2  establish  the  purpose(s)  of  each  option 

3.3  elicit  evaluation  questions 

3.4  identify  information  requirements 

3.5  design  evaluation  procedures 
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4.  Recommend  option(s)  for  evaluation 

4.1  identify  components  and  objectives/effects  to  be  studied 

4.2  determine  resources  required 

4.3  produce  terms  of  reference 


For  further  elaboration  of  evaiuability  assessment,  the  reader  is  referred  to  the  following  publications. 

Jung,  S.  M.  and  Schubert,  J.G.  Evaiuability  Assessment:  ATwo-Year  Retrospective. 
Educational  Evaluation  and  Policy  Analysis,  1983,  5(4),  435-444. 

Rutman,  L.  Planning  Useful  Evaluations:  Evaiuability  Assessment. 

Beverly  Hills:  Sage  Publications,  1980. 

Rutman,  L.  and  Mowbray,  G.  Understanding  Program  Evaluation.  Beverly  Hills: 
Sage  Publications,  1983. 

Wholey,  J.  S.  Evaiuability  Assessment.  In  L.  Rutman  (Ed.)  Evaluation  Research 
Methods:  A  Basic  Guide.  Beverly  Hills:  Sage  Publications,  1977,  41-56. 


